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Abstract. This abstract describes a research effort to apply technological advances in virtual 
reality simulation and computer-based games to create behavioral modification programs for 
individuals with Autism Spectrum Disorder (ASD). The research investigates virtual social skills 
training within a 3D game environment to diminish the impact of ASD social impairments and to 
increase learning capacity for optimal intellectual capability. Individuals with autism will encounter 
prototypical social contexts via computer interface and will interact with 3D avatars with 
predefined roles within a game-like environment. Incremental learning objectives will combine to 
form a collaborative social environment. A secondary goal of the effort is to begin the research 
and development of virtual reality exercises aimed at triggering the release of neurotransmitters 
to promote critical aspects of synaptic maturation at an early age to change the course of the 
disease. 


1.0 INTRODUCTION 

Autism is one of a group of disorders known 
as autism spectrum disorders (ASD’s). It is 
characterized by developmental disabilities 
that cause substantial impairments in social 
interaction and communication and the 
presence of unusual behaviors and 
interests. It begins before age three and 
lasts throughout a person’s life. Autism 
occurs in all racial, ethnic, and socio- 
economic groups. It is also on the rise. New 
research indicates a possible mitigation 
strategy for autism. Deletions of genes or 
regions potentially involved in regulation of 
gene expression, suggests that defects in 
activity-dependent gene expression may be 
a cause of cognitive deficits in patients with 
autism. Therefore, disruption of activity- 
related synaptic development may be one 
mechanism common to at least a subset of 
seemingly heterogeneous autism-associated 
mutations. If the above hypothesis is true, 
then controlled environmental experiences 
coupled with calculated experiential 
exposure might be able to allow treatment 
through behavioral modification to facilitate 
learning in normal environments. But given 
the characteristics of autistic individuals, 
controlled environmental experiences are 
difficult to conduct. Reports from teachers, 
therapists, researchers and parents indicate 


that many children with ASD show an affinity 
for computers. Current research indicates 
that computer-based tasks can motivate 
people with autism and encourage learning. 
Efforts to incorporate the ability to interact 
with and control virtual characters (avatars) 
within a computer generated environment 
are increasing. While there is good evidence 
that virtual environments are well accepted 
by individuals with ASD and of potential 
benefit to them, the use of the technology 
remains relatively unexplored. There is great 
potential to re-purpose technology and 
simulation content developed for the U S. 
military that combines 3D video game 
technology with the constructivist principles 
of coaching, scaffolding and deliberate 
practice to help teach cultural awareness 
and non-verbal communications skills. 

This paper describes a research effort 
designed to leverage this military 
technology, re-purpose game assets and 
adapt learning strategies to support virtual 
social skills training within a computer game 
environment in an effort to diminish the 
impact of social impairments on the lives of 
people diagnosed with ASD. If successful, 
the potential return on investment is 
enormous, both in actual cost savings, and 
in reduction of family suffering. Researching 
this area is an ethical imperative. 
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2.0 AUTISM 



Autism is one of a group of disorders known 
as Autism Spectrum Disorders (ASDs). They 
include Autistic Disorder, Pervasive 
Developmental Disorder - Not Otherwise 
Specified (PDD-NOS, including atypical 
autism), and Asperger Syndrome. These 
conditions all have some of the same 
symptoms, but they differ in terms of when 
the symptoms start, the severity of the 
symptoms, and the exact nature of the 
symptoms. The three conditions, along with 
Rett Syndrome and childhood disintegrative 
disorder, make up the broad diagnosis 
category of pervasive developmental 
disorders. ASD begins before the age of 3 
and lasts throughout a person’s life. It 
occurs in all racial, ethnic, and 
socioeconomic groups and is four times 
more likely to occur in boys than girls. [1] 


3.0 THE COST 



Dr. Michael Ganz, MS, PhD, Assistant 
Professor of Society, Human development 
and Health at the Harvard School of Public 
Health, and respected expert on the societal 
costs associated with autism and its related 
disorders claims that Autism is a very 
expensive disorder costing upwards of $35 
billion in direct (both medical and 


nonmedical) and indirect costs to care for all 
individuals diagnosed each year over their 
lifetimes. [2] 

In a paper published in the Archives of 
Pediatric Adolescent Medicine, Dr. Ganz 
details the substantial costs resulting from 
lifetime care and lost productivity of 
individuals with autism, their caretakers and 
society in general. [3] Direct costs measure 
the value of goods and services used and 
indirect costs measure the value of lost 
productivity due to autism. Physician and 
other professional services, hospital and 
emergency department services, drugs, 
equipment and other supplies, and medically 
related travel and time costs are typical 
components of direct medical costs. Special 
education, transportation, childcare and 
babysitting, respite care, out-of-home 
placement, home and vehicle modifications 
and supported employment services are 
typical components of direct nonmedical 
costs. Indirect costs are the value of lost or 
impaired work time (income), benefits, and 
household services of individuals with 
autism and their care givers because of 
missed time at work, reduced work hours, 
switching to a lower-paying but more flexible 
job, or leaving the workforce. Behavioral 
therapies, which are the largest component 
of direct medical costs, make up 6.5% of 
total discounted lifetime costs. Those costs, 
combined with very limited to non-existent 
income for their adult children with autism 
combined with potentially lower levels of 
savings because of decreased income and 
benefits while employed, may create a large 
financial burden affecting not only those 
families but potentially society in general. 

4.0 AUTISM IS INCREASING 

The U.S. Center for Disease Control and 
Prevention (CDC) states that it is clear that 
more children than ever before are being 
classified as having ASDs, however, it is 
unclear how much of this increase is due to 
changes in how we identify and classify 
ASDs in people, and how much is due to a 
true increase in prevalence. By current 
standards, ASDs are the second most 
common serious developmental disability 
after mental retardation/intellectual 
impairment. The impact of having a 
developmental disability is great for families 
affected and for the community services that 
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provide intervention and support for these 
families. It is important that we treat 
common developmental disabilities, and 
especially ASDs, as conditions of urgent 
public health concern, do all we can to 
identify children’s learning needs, and start 
intervention as early as possible to give all 
children the chance to reach their full 
potential. The CDC also states that ASDs 
can often be detected as early as 1 8 months 
and children in high-risk groups - children 
with a parent or sibling with an ASD - should 
be watched particularly closely. Studies 
have shown that among identical twins, if 
one child has autism, then the other will be 
affected about 75% of the time. In non- 
identical twins, if one child has autism, then 
the other has it about 3% of the time. Also, 
parents who have a child with an ASD have 
a 2% - 8% chance of having a second child 
who is also affected. [4], [5] 


5.0 RESEARCH DIRECTIONS 



New research conducted by Eric M. Morrow 
et al., of the Division of Genetics at 
Children’s Hospital Boston and Harvard 
Medical School, states that the regulation of 
expression of some autism candidate genes 
by neuronal membrane depolarization, 
suggests the hypotheses that neural activity- 
dependent regulation of synapse 
development may be a mechanism common 
to several autism mutations. Early brain 
development is driven largely by intrinsic 
patterns of gene expression that do not 
depend on experience-driven synaptic 
activity. Postnatal brain development 
requires input from the environment that 
triggers the release of neurotransmitters and 
promotes critical aspects of synaptic 
maturation. During this process, neural 
activity alters the expression of hundreds of 
genes, each with a defined temporal course 
that may be particularly vulnerable to gene 


dosage changes. The connection between 
experience-dependent neural activity and 
gene expression in the postnatal period 
forms the basis of learning and memory, and 
autism symptoms typically emerge during 
these later stages of development. This 
finding that deletions of genes regulated by 
neuronal activity or regions potentially 

involved in regulation of gene expression in 
autism suggests that defects in activity- 
dependent gene expression may be a cause 
of cognitive deficits in patients with autism. 
Therefore, disruption of activity-related 
synaptic development may be one 

mechanism common to at least a subset of 
seemingly heterogeneous autism-associated 
mutations. [6] 

If the above hypothesis is true, then 
controlled environmental experiences 
coupled with calculated experiential 

exposure might be able to allow treatment, 
behavioral modification, and learning to 
occur in normal environments. 


6.0 THE USE OF COMPUTERS AND 
VIRTUAL REALITY 



Reports from teachers, therapists, 
researchers and parents indicate that many 
children with ASD show an affinity for 
computers. Previous research has shown 
that computer-based tasks can motivate 
people with autism and encourage learning 
[7], The social and communication deficits of 
ASD make it difficult to engage in social 
interaction, and therefore, access to learning 
opportunities in these social settings is 
limited. Computer-based experiences in 
constructed social environments mitigate 
this deficiency. New research efforts 
incorporate the ability to interact with and/or 
control virtual characters (avatars) within a 
virtual environment. The Authorable Virtual 
Peers (AVP) program at Northwestern 
University uses language-based avatars to 
enable children diagnosed with ASD to learn 
about language and social interactions 
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through collaborative storytelling [8], The 
use of avatars has also been found to 
increase facial recognition, emotion 
recognition and social interaction skills for 
children with ASD through repeated practice 
of multiple different interactions [9], This 
affirms the commonly used approach of 
repeated practice in a natural setting to 
successfully teach skills to those with ASD. 

• Virtual reality uses sight and sound more 
that touch: auditory and visual stimuli 
have been found to be most effective in 
teaching abstract concepts to people with 
autism. 

• In the virtual environment, input stimuli 
can be modified to a tolerable level. 

• The environment can be altered 
gradually to teach generalization and 
cross-recognition. 

• Virtual reality offers a safe learning 
environment in which the individual may 
make mistakes that might be physically 
or socially hazardous in the real world. 

A research study on autistic children 
conducted at the University of Haifa, 
focused on the transfer of skills mastered 
within a virtual environment to the real world 
and found that the intelligence level or 
severity of the autism does not affect the 
ability to understand the system and 
therefore is an important way to improve 
their cognitive and social abilities [11], Six 
autistic children, ages 7-12 spent one month 
learning how to cross virtual streets, to wait 
for the virtual light at the crosswalk to 
change, and to look left and right for virtual 
cars using a simulation programmed by 
Yuval Naveh. The children in the study 
showed substantial improvement throughout 
the learning process. At the beginning of the 
study, the average child was able to use the 
2 nd level of the software, while by the end; 
they mastered the 9 th level, which is 
characterized by more vehicles traveling at a 
higher speed. A local practice area with a 
street and crosswalk, complete with traffic 
signals, was used for validation. The 
children’s ability to cross the street safely 
was tested in this area, evaluating for 
example, whether they stopped to wait on 
the sidewalk or waited for a green light 
before crossing. The children were brought 
to the practice area before and after their 
virtual learning. Here too, the children 
exhibited an improvement in their skills, 


following the training on the virtual street, 
with three of the children showing 
considerable improvement. One of the study 
participants, a 16 year old, had participated 
in the past in a road safety program in the 
school, but he was not able to learn how to 
cross the street safely. Following learning 
the skill in a virtual environment, he learned 
how to stop on the sidewalk before stepping 
into the street, to look at the color of the 
traffic light, to cross only when the light was 
green and to cross without waiting too long. 

7.0 LEVERAGING CURRENT MODELING 
AND SIMULATION TECHNOLOGY 

While there is good evidence that virtual 
environments are well accepted by 

individuals with ASD and of potential benefit 
to them, the use of this technology remains 
relatively unexplored. New computer-based 
game technologies increasingly integrate a 
social as well as a cognitive component. 
There is potential to leverage this 
technology in an innovative new direction to 
provide a context that can scaffold social 
interactions and communications skills for 
children with ASD. 

The ability to simulate, test and assess 
cognitive and social skills within a virtual 
environment provides professionals with 
rigorous practice and guidance to increase 
their chances of success in situations that 
may not be safe or cost effective to perform 
in a live training environment. 

Simulations and games can supplement 
traditional training methods by providing 
challenges and experiences that closely 
approximate a complex situation in the real 
world where students must think in real-time 
and the course of events will be determined 
by their decisions. Students with ASD may 
be able to develop a deeper understanding 
of the knowledge presented and retain that 
information better when it is learned through 
the process of repeatedly solving problems 
in realistic situations. This approach places 
the learner in a “real-world” environment, 
which allows the student to learn in context 
and apply what they have learned. It is this 
contextual experience of knowledge 
acquisition in an authentic environment that 
facilitates the learner to create their own 
constructs that can be applied to new and 
unfamiliar situations. There is also an 
opportunity to provide practical, hands-on 
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experience in situations that cannot easily 
be practiced using real scenarios. 

There is also great potential to re-purpose 
technology and simulation content 
developed for the U.S. military that 
combines 3D video game technology with 
the constructivist principles of coaching, 
scaffolding and deliberate practice to help 
teach cultural awareness and non-verbal 
communications skills. The emerging 
importance of cultural identity and its 
inherent frictions make it imperative for 
soldiers and leaders to understand societal 
and cultural norms of the populaces in which 
they operate and function. Much of this 
communication occurs through nonverbal 
channels, especially when language skills 
are minimal or absent. It is totally feasible to 
leverage this military technology, re-purpose 
game assets, and adapt learning strategies 
to support virtual social skills training within 
a 3D video game in an effort to diminish the 
impact of social impairments on the lives of 
people diagnosed with ASD. 


8.0 SPECIFIC APPLICATION STRATEGIES 



The symptoms and characteristics of autism 
can present themselves in a wide variety of 
combinations. The uniqueness of each 
individual with autism and the context of 
their lives provide interesting design 
challenges for the successful creation and 
adoption of technologies for this domain. 
The first goal is to enable children with ASD 
to not only interact with a virtual 
environment, but also to build social skills. A 
socially relevant scenario can be designed 
which encourages human interaction with 
artificially intelligent avatars. Inside the 
virtual world, which includes settings 
commonly encountered in everyday life such 
as restaurants, shops, offices, parks and 


other social places, autistic individuals will 
be able to interact with other real people’s 
avatars as practice. The user interface and 
scripts will be extremely clear and simple, 
and since previous ASD research has 
shown benefits of storytelling, each game 
scenario may consist of a short vignette 
design to elicit response from the student. 
Vignettes used for the project may include: 

• Teaching the student how to interact 
through social stories, modeling, role- 
playing and other activity-based learning. 

• Conflict resolution and managing 
disagreement with compromise and 
recognizing the opinions of others. 
Learning not to respond with aggression 
or immature mechanisms. 

• Turn taking and other socially acceptable 
mannerisms such as verbal interactions, 
changing conversational topics, 
introductions to new people and others. 

Individuals with autism will encounter 
prototypical social contexts via a computer 
interface and will have to interact with 3D 
avatars within the game that have 
predefined roles, tasks, and visible body 
language. Within the context of the game’s 
scenario, the student will trigger events, 
which equate to learning objectives that they 
must successfully interact with to advance 
further into the game. Incremental learning 
objectives will eventually combine to form a 
fully collaborative social environment. The 
initial sequence of tasks will most likely 
follow a linear model progressing from 
simple tasks to more complex ones. The 
effects of the student’s actions, will impact 
the behaviors of the other avatars within the 
scenario in a realistic fashion. Further, as 
the scenario reacts to the student’s input, it 
will track performance and provide feedback 
concerning the consequences of particular 
actions and or omissions. In terms of people 
with ASD learning social behaviors, errors 
do need to be made to support learning. 
Therefore, a balance must be made 
between allowing the user to make errors 
and clearly showing what options are 
available at any given time within the game. 

A secondary goal is to begin the research 
and development of VR exercises aimed at 
triggering the release of neurotransmitters to 
promote critical aspects of synaptic 
maturation at an early age, to change the 
course of the disease. Rigorous scientific 
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evaluation is necessary to estimate the likely 
benefits of this approach and its application 
to the individual. Research should also 
attempt to evaluate the contribution of this 
technology to any observed gains through 
comparison with traditional teaching 
approaches. 

Web-based instruction and testing via 
“intelligent" computer simulations of typical 
social environments will prove an efficacious 
means for people with ASD to acquire social 
skills. Computer-based media allows people 
with autism continuous access to the 
curriculum, while concurrently allowing 
researchers to track the frequency of 
exposure and/or duration of exposure to a 
given skill (e.g. time logged onto vignettes 
and testing). It is expected that participants 
may engage in virtual skills training for 
longer durations when compared to 
traditional lecture-based curriculum. 

For the parents, teachers and families of 
autistic children, understanding and active 
participation can be critical to their 
development and eventual independence. 
Adolescents often play games. These 
games, whether played in isolation or in a 
group setting, may be an effective 
reinforcement for skills that can be practiced 
in the home with family members and peers. 
Combining skill instruction with the gaming 
experience offers the gamer a chance to 
repeatedly practice skills. Although the skills 
learned may be constrained to the 
capabilities of each individual, such 
strategies may prove to be the gateway to 
increased socialization and acceptance by 
peers. 

The vision is to use online, PC-based games 
and immersive 3D environments that 
leverage existing DOD research and 
development in modeling, simulation, 
serious gaming, performance assessment, 
and after action review technologies. The 
goal is to establish a seamless management 
and delivery capability to provide a 
distributed virtual environment where skills 
can be practiced and honed as a student 
interacts within each prescribed scenario. 
Virtual reality and gaming applications for 
social skills may prove to be less resource 
intensive than traditional in-vivo and “Video 
Self Modeling” training models. Further, this 
technology may provide a more engaging, 
and socially controlled environment in which 


autistic individuals can practice social skills 
without excessive distractions. Attention 
must be paid to new collaborative 
technologies such as massive multi-player 
environments that allow interactive 
experiences for groups as well as 
individuals. Automated support tools should 
be investigated to help teachers perform in 
depth assessments of student performance 
and to identify and mitigate critical behaviors 
by providing essential feedback. In this way, 
distributed and collaborative virtual 
environments can be incorporated into the 
continuum of ASD treatment to work in 
tandem with the full spectrum of other case 
management interventions [11]. 

Based on positive outcomes from this effort, 
additional studies could also begin to look at 
which social skills acquired and practiced via 
virtual models will generalize to school, 
home, and community environments. 
Scenario authoring capabilities should also 
be investigated to provide the ability to 
modify and insert new resources into the 
virtual environment as required by the 
various ASD treatment interventions. The 
ability to tailor virtual scenarios to specific 
student needs is intended to help the 
student contextualize social situations and 
events. The application of this augmented 
virtual reality technology contained within 
the science of modeling and simulation 
could produce tremendous synergy in 
mitigating the treatment and educational 
interventions to reduce the rising cost in 
resources, as well as in pain and suffering. 
Researching this area is an ethical 
imperative. 
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